Local Polynomial Models for Classification∗

Local likelihood approaches to statistical inference attempt to construct nonparametric estimators based on local polynomial fits to the likelihood function at a given point of interest. As opposed to approaches based on the construction of complex global models which are then used in order to predict future behavior, the local models fit a set of parameters to a simple local model in the vicinity of a test point of interest. We consider a complexity regularized approach to local maximum-likelihood fitting, and show that it is related to support vector machines. While our results are applicable to both regression and classification, we focus on the latter case and establish the consistency of the approach for the holdout method. Finite sample performance bounds are also given. Experimental evidence is provided to demonstrate the flexibility of the method. This includes an algorithm for selecting favorable local metrics for k nearest neighbor classifiers.